Classifiers are used in many systems to take a data set and generate one or more conclusions or classifications based on the data set. For example, a data set may be classified as indicating a fraudulent transaction after being passed through a classifier, or medical information can be passed through a classifier that then indicates the probability that a patient has a certain condition. Many of these systems may employ machine learning to tune the classifier. This requires the provisioning of a training set of data that has known results.
In many machine learning systems in general, and binary classification in particular, it is difficult understand what the reason is for a particular output classification to be for example, positive rather than negative. For example, in a fraud detection system, a human reviewer might want to know the reason or reasons an automatic fraud detection system labeled a user or a transaction as fraudulent. The current solution is to manually analyze the classification model and understand what lead to the observed output.